Approval-rating systems that never reward insincerity

Electronic communities are relying increasingly on the results of continuous polls to rate the effectiveness of their members and offerings. Sites such as eBay and Amazon solicit feedback about merchants and products, with prior feedback and results-to-date available to participants before they register their approval ratings. In such a setting, participants are understandably prone to exaggerate their approval or disapproval, so as to move the average rating in a favored direction. We explore several protocols that solicit approval ratings and report a consensus outcome without rewarding insincerity. One such system is rationally optimal while still reporting an outcome based on the the usual notion of averaging. That system allows all participants to manipulate the outcome in turn. Although multiple equilibria exist for that system, they all report the same average approval rating as their outcome. We generalize our results to obtain a range of declared-strategy voting systems suitable for approval-rating polls. 1 Approval ratings and their aggregation Approval ratings are one mechanism that communities can use to offer incentive and reward for good behavior or service. The prospect of feedback following a given interaction presumably increases the accountability of that interaction for all parties involved. Publication of approval ratings then enables appropriate consequences to follow from positive or negative experiences. In this paper we consider several forms of aggregation and we show that some methods can reward insincerity while others cannot. We next provide several examples of approval rating systems and formulate a general form of an approval rating poll. 1.1 Examples of approval rating polls Subscribers and observers of media frequently learn of the results of approval rating polls that attempt to discern how strongly a participating electorate endorses a person or a position of interest. As an example, several web sites post varous forms of approval ratings for films and games. Specifically, Rotten Tomatoes [2] posts the results of two polls for each film: • In effect, each review is turned into a 0 or 1 value, and the Tomatometer is the average of those values expressed as a percentage. Putative viewers might consult a film’s Tomatometer value to determine whether they should see that film. • Each critic can also rate a film’s overall quality on a 1–10 scale. Rotten Tomatoes then publishes the average of all such ratings. Finally, consider the electronic marketplace, in which participants are asked to rate the honesty and effectiveness of merchants and customers. Sites such as eBay poll their participants concerning how strongly they approve of the behavior of the marketplace members they encounter in transactions. Upon completion of a transaction, the involved parties are asked to rate each other. An aggregation of an indvidual’s approval ratings is posted for public view, so that members can consider such information before engaging that individual in a transaction. 1.2 Formulation We next define a general instance of an approval rating poll to facilitate presentation of our results. • An electorate of n participants is polled. Based on the participants’ response and the aggregation protocol at hand, the result of the poll will be published as a rational number in the interval [0, 1]. • Each participant i has in mind a sincere preference rating ri, where 0 ≤ ri ≤ 1, that can be construed as that participant’s dictatorial preference. The tuple of all participants’ sincere ratings is denoted by the vector ~r. We further make the reasonable assumption that voter i’s preferences are single-peaked and non-plateauing. • Finally, voter i participates in the poll by expressing a rating preference of vi, which may or may not be the same as ri. In fact, we are particularly interested in situations where vi 6= ri. For example, consider an eBay customer who undertakes a transaction with a highly approved merchant. If the customer becomes disgruntled with the merchant, then the customer’s resulting rating of the merchant might be overly negative, precisely because of the merchant’s otherwise high rating. The tuple of all expressed approval ratings is denoted by the vector ~v. This paper considers an approach that can account for, mitigate, or prevent the use of insincerity to increase a participant’s effectiveness in an approval rating poll. 1.3 Aggregating approval ratings The results of an approval rating poll are typically reported by an aggregation procedure that is disclosed a priori. In this section, we consider two popular aggregation schemes: average and median. Average aggregation Here, the result of the approval rating poll is computed as the average of the participants’ expressed approval ratings: v̄ = n j=1 vj n . While the Average aggregation function is sensitive to each voter’s input, it has an important disadvantage: Voters can often gain by voting insincerely. For example, the 1983 film Videodrome has five critics’ ratings on Metacritic [1]. If we assume that these critics rated the film sincerely (that each would prefer that the average rating of the film be his or her rating), we have ~r = [0.4, 0.7, 0.8, 0.8, 0.88]. If these preferences are actually expressed sincerely in an Average aggregation context, then we have ~v = ~r and the Average outcome is 0.716. Consider voter 5, whose ideal outcome is r5 = 0.88. That voter could achive a better outcome by not expressing the sincere preference v5 = 0.88 and instead voting v5 = 1. The resulting Average aggregation yields the outcome 0.74, which, being closer to 0.88, is preferred by voter 5 to 0.716. Median aggregation (n odd) Another possible aggregation function computes a median of ~v: ṽ is a value that satisfies |{i : ṽ < vi}| ≤ n 2 ≤ |{i : ṽ ≤ vi}|. 2 According to the median For the applications we describe, it is reasonable to assume that each voter i would prefer that the outcome be as near to the ideal ri as possible. This single-peaked assumption makes possible the optimal strategy we describe in section 2. The above definition does not necessarily prescribe a unique outcome when n is even; we address this issue below. voter theorem [4, 10], when n is odd, Median aggregation becomes the unique, Condorcetcompliant [13] rating system, yielding a result that is preferred by some majority of voters to every other outcome. Unfortunately, Median aggregation can effectively ignore almost half of the voters— majority rule can mean majority tyranny. Given the tuple of votes ~v = [0, 0, 0, 1, 1], the 1voters are effectively ignored when the median, 0, is chosen as the outcome. Majority tyranny could be quite undesirable for polls of this type, especially when the goal of aggregating ratings is to represent a satisfactory consensus for all voters. The Average outcome of the above tuple, 0.4, arguably provides such a much better consensus. In contrast with Average aggregation, Median aggregation is nonmanipulable by insincere voters—at least when n is odd: a voter i can never improve the outcome from his or her point of view by voting vi 6= ri. (The treatment for an even number of voters and the proofs here and below are omitted for space; this material can be found in LeGrand [12, ch. 3].) Thus, Median aggregation does not reward insincerity for an odd number of participants. Without losing nonmanipulability, the Median function can be generalized to give the outcome ṽ where |{i : ṽ < vi}| ≤ bn ≤ |{i : ṽ ≤ vi}| for any 0 ≤ b ≤ 1 (in this notation, the b is intended as a parameter modifying the tilde symbol). If bn is an integer, there may be more than one 0 ≤ φ ≤ 1 that satisfies |{i : φ < vi}| ≤ bn ≤ |{i : φ ≤ vi}|. In that case, define Φ as the set of all such φ. Then